Conditions for Stability and Convergence of Set-Valued Stochastic Approximations: Applications to Approximate Value and Fixed point Iterations with Noise

The study of stochastic approximation algorithms (SAAs) with setvalued mean-fields has been popular during recent times due to their applications to many large scale model-free problems arising in stochastic optimization and control. The analysis of such algorithms requires the almost sure boundedness of the iterates, which can be hard to verify. In this paper we extend the ideas of Abounadi, Bertsekas and Borkar involving regular SAAs (with point-to-point maps), to develop easily verifiable sufficient conditions, based on Lyapunov functions, for both stability and convergence of SAAs with set-valued mean-fields. As an important application of our results, we analyze the stochastic approximation counterpart of approximate value iteration (AVI), an important dynamic programming method designed to tackle Bellman’s curse of dimensionality. Our framework significantly relaxes the assumptions involved in the analysis of AVI methods. Further, our analysis covers both stochastic shortest path and infinite horizon discounted cost problems. Generalizing further, we present an SAA, together with an analysis of it’s stability and convergence for finding fixed points of contractive set-valued maps. To the best of our knowledge ours is the first SAA for finding fixed points of set-valued maps. email:[email protected] email:[email protected]